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ABSTRACT 



Research on factors related to student academic achievement 
typically involves the use of some type of total cognitive score as the final 
outcome measure. However, in using a total score, it is assumed that 
variables related to student learning have the same effect on student 
learning across academic skill levels. Using a subsample from the first 
followup to the National Education Longitudinal Study of 1988 (NELS:88), this 
study examined the differential effects of between- and within- school 
indicators on the learning of mathematics at different skill levels. Data was 
taken from students whose mathematics teachers and school administrators 
participated in the NELS:88 school survey, for a final student sample of 
4,848. Indicators were chosen from data for the student and teacher surveys. 
The study offers evidence that the relationship between education indicators 
and academic learning differs depending on the level of skill under 
investigation. When studying the effects of various inputs and processes of 
schooling on student learning, the researcher should move away from the 
strict use of total scores and consider the use of scores that break down 
student achievement into learning at various levels. An appendix discusses 
the structural equation model used for the analysis. (Contains 3 figures, 15 
tables, and 46 references.) (Author/SLD) 



* Reproductions supplied by EDRS are the best that can be made 

* from the original document . 



Probability of Proficiency I 



The Use of Probability of Proficiency Scores in the Study of Relationships Between 



CN 

r- 

m 

Os 

O 



s 



Education Indicators and Student Mathematics Learning at Different Skill Levels 



Pamela R. Elliott 



Clinical and Social Psychology 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL 
HAS BEEN GRANTED BY 

B&uO* 2JAic_th_ 



University of Rochester 



685 South Avenue 



Rochester, NY 14620 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 



716-262-2920 



pelliott@psych.rochester.edu 



1 1 q department of education 

Ot^cfof D EduMhonal Research and Improvement 

EDUCATIONAL resources information 
CENTER (ERIC) 



originating it. 

□ Minor changes have been made to 
improve reproduction quality. 



Points of view or opinions stated in this 
document do not necessarily represent 
official OERl position or policy. 



March 16, 1997 



Paper presented at the 1997 meeting of the American Educational Research Association. Chicago 



O' 

vS 

OS' 

O 




BEST COPY AVAILABLE 



2 



* • 

Probability of Proficiency 2 

\ ’ 



Abstract 

Research of factors related to student academic achievement typically involves the use of some 
type of total cognitive score as the final outcome measure. However, in utilizing a total score, 
one assumes that variables related to student learning have the same effect on student learning 
across academic skill levels. Utilizing a sub-sample of the first follow-up of the National 
Education Longitudinal Study of 1988, the primary objective of this work was to examine the 
differential effects of between- and within-school indicators upon the learning of mathematics at 
different skill levels. The current study offers evidence that the relationship between education 
indicators and academic learning differ, depending upon the level of skill under investigation. 
Therefore, when studying the effects of various inputs and processes of schooling upon student 
learning, one should move away from strictly using total scores, and consider the use of scores 
that break down student achievement into learning at different skill levels. 
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Introduction 



One of the six national goals states: “By the year 2000, American students will leave 
grades four, eight, and twelve having demonstrated competency over challenging subject matter, 
including ... mathematics. (US. Department of Education, 1990. P 5). Related to this goal, 
mathematics reformers suggest that the intent of mathematics instaiction should include to 
promote learning across a broad range of mathematical topics, and to promote learning of 
mathematical thinking, reasoning, and problem-solving skills (e.g. American Association for the 
Advancement ofScience, 1989, National Council ofTeachers in Mathematics, 1989, National 
Research Council, 1989). The question we are left to answer is “How do we most effectively 
promote the learning of mathematics?’' Much of the research on educational attainment concludes 
that the effects of educational inputs such as per pupil spending, teacher experience, and teacher 
educational background are unimportant predictors of education outcomes. Additionally, the 
impact of an educational input is inconsistent across studies. Moreover, research suggests that 
individual and family background information explain most of the variation in student achievement 
(Hanushek, 1986). 

In order to ensure that the most is being done to improve the learning of mathematical 
skills among our nation’s youth- including mathematics thinking, reasoning, and problem-solving 
skills, we must engage in the rigorous study of factors that affect learning. The current paper 
suggests that this should be done via the utilization of criterion-referenced scores such as 
probability of proficiency scores. Proficiency scores allow one to asses how well as student has 
mastered concepts within a given skill category, while the probability of proficiency score offers 
the probability that a student is proficient at a given skill level. Research of factors related to 
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student learning typically involves the use of some type of total cognitive score as the final 
outcome measure. However, in utilizing a total score, one assumes that variables related to 
student learning are have a homogeneous affect on learning across academic skill levels. This 
study purports to show that the relationship between education indicators and academic learning 
differ, depending upon the specific skill level under investigation. 

Method 

For this paper, multilevel structural equation modeling was used to estimate the 
multilevel models of mathematics proficiency The use of multilevel structural equation modeling 
allows one to capture the multilevel, organizational nature of schooling, including the 
interrelationships amongst indicators within and across levels of the system (Darling-Hammond. 
1992; Oakes, 1986; Porter, 1986; Willms, 1992). It also affords the decomposition of the 
relationships between variables of the model into direct, indirect, and total effects of variables 
within and between levels of the system. Finally, multilevel structural equation modeling allows 
for a measurement model to be specified, affording the inclusion of latent variables (for a 
discussion of multilevel structural equation modeling, see Appendix of this paper, and see Kaplan 
& Elliott, 1997; for a brief explanation of parameter estimation, see Kaplan & Elliott, 1997, for 
detailed information about parameter estimation see Muthen, 1994, and for details on software 
implementation, see Bentler, 1989; Kaplan & Elliott, 1997; Nelson & Muthen, 1991). 
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Data Source 

The data for this research comes from the public release files of the First Follow-Up 
Study of the National Education Longitudinal Study of 1988, commonly referred to as NELS:88 



Probability of Proficiency 5 



(NCES, 1988). NELS.88 is an ongoing study of a national sample of students as they progress 
from the eighth grade through high school, and onto postsecondary education and / or work 
(NCES, 1995) Its primary purpose is to collect policy relevant information concerning such 
areas as the effectiveness of schools, curriculum paths, special programs, and curriculum content 
and exposure NELS 88 collects information on family background, student aspirations and 
attitudes, experiences in and out of school, and high school transcripts It also contains data from 
students' teachers, schools, and parents. The base year data was collected in 1988, when the 
students were in 8th grade. A follow-up study is done every two years. This study utilizes the 
first follow-up study, when the sampled students were in the tenth grade. 

The subset of data used in this research was obtained as follows. Of the 20,706 
students appearing in the first follow-up study, only those students whose mathematics teachers 
participated in the teacher survey and whose school administrator participated in the school 
survey were retained. Next, indicators were chosen for the within-school model, consisting of 
data from the student and teacher survey. After listwise deletion of any missing data and multiple 
response, the final student level sample size used in the development of the within-school model 
was N=4848. 

Once the within school model was estimated, the between-school model was 
developed The data for this model was obtained by merging the within-school data with 
indicators to be used in the between-school model. Only those schools with at least 10 students 
were retained. After listwise deletion based on the between-school variables, the between-school 
sample was G=1 1 1, and the within-school sample was reduced to N=1504 (I 504 students within 
1 1 1 schools). 
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Because of the sampling design of NELS:88, there were too few students within 
classrooms to estimate a three level student within classroom within school model Therefore, the 
multilevel model was conceptualized as a two level model consisting of a within-school level and 
a between-school level. Several teacher level variables appear in the within-school model, 
because those teacher level indicators were hypothesized to vary within schools. The author 
realizes that this may introduce some bias in the estimation results, however, it was important 
theoretically to have the teacher-level variables in the model. Moreover, since the ratio of 
teachers to students was approximately one-to-one, the bias should be at a minimum 

Selection of Indicators 

Selection of the indicators for the within- and between-school models was guided by 
the education indicators literature (see e g. Catterall, 1989; Murnane and Raizen, 1988; Oakes, 
1989; Shavelson, Webb, & Rowley, 1989; for a review of the indicators literature, see Elliott, 
1996). However the choice of indicators was limited to data available from the public release files 
of the first follow-up of NELS88. Indicators of the within- and between-school models consist of 
a combination of single item indicators, and scales which were developed via maximum likelihood 
exploratory factor analysis (MLEFA).' 

Within-School Indicators 

The within-school indicators are presented in Table 1. Within-school indicators at the 
teacher level include (see Table 1 for details on coding): GRADING, a scale measuring the 

In each case a variety ol alternative factor structures with oblique rotation were explored. The 
criterion lor choosing the number oi factors was based on the change in chi-square goodness of 
lit when the number ol factors changed as well as substantive significance and interpretability of 
the factors. Scales were lormed via unit weighting of the variables defining the factor. 
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emphasis the teacher gives to different areas when setting student grades, YRSEXP 
the number of years of secondary teaching; ENRICH, a scale consisting of responses to six 
questions related to the teacher’s participation in enrichment; and SIZE: the number of students 
enrolled in the mathematics course. 

Student-level within-school indicators include (see Table 1 for details on coding): 
HOMEWORK: the amount of time the student spends in and out of school on homework each 
week; CLUB: the student’s extent of participation in academic clubs; PROBLEM: the student’s 
perception of the teacher’s emphasis on teaching problem solving skills, EFFORT: how often the 
student tries as hard as he or she can in mathematics class; REQUIRED: a dichotomous indicator 
of whether the current mathematics class is required or an elective (0=required), PRIOR a 
measure of the student’s prior mathematics grades; and HANDS: a scale measuring how often the 
student uses computers, hands-on materials, models, and calculators in mathematics class. 

The final outcome measure of the within-school model is MATH: the probability of 
mathematics proficiency This outcome is calculated at each of five mathematics skills levels 
wherein MATH I is the probability of mathematics proficiency at Level 1 , defined by “single step 
operations which rely on rote memory,” MATH2 is the probability of mathematics proficiency at 
Level 2, defined by "simple operations with decimals, fractions, powers, and roots." MATH3 is 
the probability of mathematics proficiency at Level 3, defined by "simple problem solving, using 
low level mathematical concepts." MATH4 is the probability of mathematics proficiency at Level 
4, defined by "understanding intermediate level mathematical concepts and the use of multi-step 
solutions. MATH5 is the probability of mathematics proficiency at Level 5defined by "being able 
to solve multi-step complex problems." Since MATH is calculated at each of the five skill levels, 
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each student has five probability of proficiency scores. These scores ofTer, for a given student, (he 
probability of proficiency at the corresponding skill level. 

Between-school Indicators 

The between school indicators are presented in Table 2, and include: LUNCH: the percent 

students receiving free or reduced lunch in the school, SALARY: average teacher salary within a 

school; MTEACHR: the number of full-time mathematics teachers; GROUPING: a dichotomous 

variable measuring if the school use homogeneous grouping for placement of tenth grade students 

in the mathematics class (0=no); DROPOUT: a dichotomous indicator of whether the school has 

0 

a dropout prevention program (0=no); STAFDEV: a measure of how much emphasis the school 
places on staff development programs, ADVMATH: the number of advanced mathematics 
courses offerred; and PRESS: a measure of the academic press of the school (see Table 2 for 
details on coding). 



Results 

Two independent models of schooling were developed as a precursor to the 
formulation of the multilevel model of schooling. A within-school model was developed to model 
variation across students. However, students are nested within schools. Therefore, a school-level 
model was developed to explain the effects of school level variables upon the average levels of tbe 
student level variables across schools. Once the separate models were developed, a multilevel 
model combining the two separate models was estimated. 

erJc 
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Results of Within-School Model 

Five within-school models were developed, using LISREL (Joreskog & Sorbom, 
1993) The first model of indicators related to probability of mathematics proficiency was 
developed using the probability of mathematics proficiency at Level 3 Level 3 was chosen as the 
outcome since the sample displayed the most variation in the probability of mathematics 
proficiency at Level 3. Once the initial model was estimated, it was assessed for goodness-of-fit 
by looking at the various fit indices offered by LISREL (Joreskog & Sorbom, 1993) including 
Chi-square, the Non-Normed Fit Index (NNFI) and Root Mean Square Error of Approximation 
(RMSEA). 

Once the initial model was assessed for goodness-of-fit, ways to improve the fit of the 
model were investigated. The approach used to modify the model was suggested by Saris, 
Satorra, and Sorbom (1987), and by Kaplan (1989, 1990), which utilizes information from both 
the Modification Index (Ml) and the Expected Parameter Change (EPC). The MI, developed by 
Sorbom (1989), measures the expected drop in the value of the chi-square statistic if a fixed 
parameter is freed in the model. The EPC provides the approximate size of a fixed parameter if it 
is freed (Saris, et al., 1987). Possible modifications to the model were scrutinized to make sure 
that they made substantive sense, and to make sure that they were in agreement with the 
underlying theoretical model of schooling as suggested by the education indicators literature. 

Once the within-school model developed with the outcome at Level 3, it was 
estimated four additional times, using each of the other probability of proficiency levels as the 
final student outcome measure. This way one can obtain information about how the factors 
included in the within-school model differentially effect the student's probability of proficiency 
across skill levels. 10 
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Assumptions underlying the use of structural equation modeling were examined 
Two of the within-school model outcomes, the probability of mathematics proficiency at Level 1 
and at Level 5 displayed high levels of skewness and kurtosis (see Table 3), violating the 
assumption of multivariate normality. These abnormalities are believed to reflect a ceiling effect 
for the case of Level I and a floor effect for Level 5, due to the nature of the mathematics 
assessment developed by NCES. The Level I items are too easy for most tenth grade students, 
therefore most of the students perform well on those items. This results in a ceiling effect at 
Level 1. For Level 5, a floor effect is exhibited because the items at Level 5 are too hard for most 
tenth grade students (a majority of tenth graders have not been exposed to Level 5 skills). Most 
tenth grade students perform poorly on those items, resulting in a floor effect at Level 5. 
Violations of the assumption of multivariate normality could inflate chi-square as well as biased 
standard errors of estimates. However, since one of the purposes of this paper is to explore the 
use of probabilities of proficiency as outcomes, it was decided not to transform the data to be 
more normally distributed. 

Results of Within-school Model. Developed Using Level 3 as Outcome 

The first model to be estimated was a within-school model with the probability of 
proficiency at Level 3 as the outcome measure (see Figure 1) Descriptive statistics of all within- 
school variables are presented in Table 3. Results of this initial model show that the model does 
not fit the data (see Table 4). 




Six modifications were made to the initial model, based upon the values of the 
modification index (MI) and expected parameter change (EPC). Those paths with high Mis along 
with high EPCs were freed if the path made substantive sense (see Table 5). Results of the final 
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model are displayed in Table 4 and Figure 2. While the model still does not statistically fit the 
data, statistical fit is much improved over that of the initial model, with a large decrease in chi- 
square, increase in NNFI, and a value of RMSEA close to 0.05. 

Next the final model was re-estimated, using the four other probability of proficiency 
levels as outcomes. As can be seen from the various fit indices, the models vary in degree of fit. 
depending upon which outcome measure is used (see Table 4). Table 5 displays the Mis and 
EPCs for each re-estimate of the within-school model developed on the probability of proficiency 
at Level 3. As evident from Table 5 the modifications made based on Level 3 would not have 
been made when using the other probability of proficiency levels as outcomes For example, the 
M3 and EPC for the first modification with Level 3 as the outcome are 94.79 and 0. 13, 
respectively. However, when the outcome is Level 1, the MI and EPC are 0 12 and 0.00, 
respectively. These values are much lower than in the previous model. Low Mis and EPCs for 
the first modification are also seen when Level 2 and Level 5 are the outcome measures. 

Results from the Mis and EPCs could possibly be interpreted as the relationship 
between the use of hands-on manipulatives and computers on the probability of mathematics 
proficiency is important for learning simple problem solving and intermediate level concepts 
(Levels 3 and Level 4, respectively). However, using hands-on materials may not be as importa.m 
for Level 1 and Level 2, where most students already have a high probability of proficiency 
anyway. At Level 5, most students have a low probability of proficiency because the items are 
very hard. Therefore, using hands-on materials may not make any difference in learning. 
Alternatively, results for Level 5 could be due to Level 5 skills not lending themselves to the use 




of hands-on materials. 
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Table 6 through Table 1 0 present the direct, indirect, and total effects of the final 
within-schoo! model developed from probability of proficiency at Level 3, and estimated with the 
other levels as outcomes 2 . Several findings emerged across all skill levels which are supported by 
previous literature on mathematics learning. For example, work, by Reynolds and Walberg (1992) 
support the result that prior achievement is a powerful predictor of current achievement Their 
work also offers supporting evidence that student effort indirectly affects achievement in 
mathematics, as mediated by such factors as prior grades. 

In comparing the direct effects of the model when'using different outcome levels, 
there are several differences worth noting. There is a moderate amount of variation in the size of 
the direct effect of HOMEWORK on MATH. When Level 1 is used, the effect is 0.062, at Level 
2 the effect is 0.067, Level 3 results in a coefficient of 0.057, Level 4 results in 0 047, and when 
Level 5 is the outcome, the coeffi cient is estimated at 0.018, a difference of 0.044 from the Level 
1 coefficient. I he amount of time a student spends on homework per week is positively related to 
one's probability of proficiency at Level 1 through Level 4. Perhaps doing homework can 
increase one's probability of proficiency at the lower proficiency levels. However, at Level 5, the 
amount of time spent on homework has a much weaker relation to the probability of proficiencv 
(being able to solve multi-step complex problems). 

Several possible explanations for this difference can be offered For example, if ok 
has a low probability of proficiency at Level 5, doing homework may not help increase one's 
probability of proficiency. Alternatively, teachers may be assigning homework in a format that 
does not foster learning of multi-step complex problems. In other words, it may not just be the 

' Due to space limitations, initial model estimates are not provided, but are available from the 
author upon request. 
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amount of homework that the student does, but more importantly, the kind of homework the 
student does. In fact, Sasser (1990/91) offers evidence that. students who receive computer 
tutorials as homework in mathematics class show higher achievement than students who receive 
traditional textbook exercises as homework (he studied 92 college freshman and sophomore 
volunteer elementary education majors in their learning of concepts of algebra). The point is that 
the strength of the relationship between amount of homework and probability of proficiency 
differs, depending on which level of probability of proficiency is used as the outcome measure. 

Variation also exists in the strength of the relationship between the use of hands-on 
materials in the classroom and the probability of mathematics proficiency across the five skill 
levels. Stronger direct effects are evident at Level 3 and Level 4, while a much weaker 
relationship is evident Level 5. The direct effect of the use of hands-on materials upon the 
learning of Level 1 and Level 2 skills are almost zero, although the use of hands-on materials does 
have a significant total effect upon the probability of learning Level 4 skills. 

Evidence supporting the effect of the use of hands-on materials, and more specifically 
computers, upon a student’s problem-solving ability (Levels 3, 4, and 5) come from a variety of 
studies. For example, Blume and Schoen (1988) found that skills developed from computer- 

usage transferred to mathematical problem-solving. Programmers used systematic trial more 

/ 

frequently than and corrected more errors in their work than non-programmers. Consistent with 
their finding is work by McCoy and Burton ( 1 988) who found that “after programming 
instruction, both the ability to use mathematical variables and mathematical problem solving 
ability significantly improved. A later study by McCoy and Dodi (1989) found that computer 
usage increased problem-solving achievement in mathematics. Finally, Damarin, Dzaik, Stull, and 
Whiteman (1988) found significant increases in problem solving ability as a result of using 
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computer based instructional materials across ninth-, tenth-, and twelfth-grades, in the 
corresponding mathematics courses. 

The next analysis involved the calculation of the intraclass correlations for each of the 
within-school variables. Intraclass correlations provide a measure of the proportion of variance in 
the within-school variables due to the existence of variation between-schools. A large intraclass 
correlation suggests the need for a multilevel analysis (see Muthen, 1991). The intraclass 
correlations ranged from 3% to 40% (see Table I I), providing evidence of between-school 
variation and therefore warranting a multilevel analysis. 

Results of Between-School Model 

Since the intraclass correlations suggested the need for a multilevel analysis, a 
between-school model was explored, to be combined with the within-school model in the 
development of the multilevel model. Descriptive statistics for the between-school variables are 
presented in Table 12 (for a diagram of the between-school model, see Figure 2). The initial 

between-school model was found to fit the data: x (10) =14.50, p > .05, N74F1 = 0.768, RMSEA 
= 0.065, P(RMSEA < 0.05) = 0.333. For the between-school model here were no violations of 
assumptions underlying structural equation modeling. Model modifications were investigated in 
the same manner-as for the within-school model-. No modifications were made, therefore the 
initial between-school model is also the final model. 

Table 13 presents the direct, indirect, and total effects of the between-school model 
There are several moderately large effects worth noting, including the total effect of LUNCH on 
PRESS (-0. 1 666) and the total effect of LUNCH on ADVM ATH (-0.2506), the total effect of 
STAFDEV on PRESS (0.2638), and the total effect of ADVMATH on MTEACFfR (0.2296). 
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Results of Multilevel Model 

The building of the multilevel model involved the combining of the two separate uni- 
level models, while allowing paths to be estimated between the between-school indicators and the 
intercepts of the within-school variables. This assumes that there exists a between-school model 
which can explain the variation in the intercepts and means of the within-school indicators across 
schools 

Five multilevel models were estimated, differing only in the final within-school 
outcome (level of probability of mathematics proficiency). To combine the between- and within- 
level models into each of the five multilevel models, the paths between PRESS and the intercepts 
and means of the indicators of the within-school model, with the exception of YEARS, SIZE, 
REQUIRED, and PRIOR, were allowed to be estimated. These four indicators (YEARS, SIZE. 
REQUIRED, and PRIOR) are assumed to be exogenous to the multilevel model (they are not 
explained by the model, but rather are inputs to the model). 

Fit statistics are presented in Table 14 3 Although the chi-square statistic suggests 
that the models do not fit the data (with possible exception of Level 5), the alternative fit indexes 
offer evidence that each of the five multilevel models fit the data, with some variation in fit across 
models. Since the alternative fit indexes offer evidence of fit, and there was no substantive 
justification to add or remove paths across the levels, the multilevel model was not modified 
beyond the initial specification. 

The standardized regression coefficients of the within-school intercepts on PRESS 
are given in Table 15. Most of these effects are small and insignificant, and some of the signs of 
the coefficients do not make substantive sense (see Table 15). The intercepts are interpreted as 
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the expected values of the indicator, given that the other indicators are zero. However, zero may 
not be an admissible value for many of these indicators. Therefore, the coefficients relating the 
intercepts to the school level variables may not be interpretable. This problem is the issue of 
centering which is related to the identification problem in the context of multilevel structural 
equation modeling (see; Bryk & Raudenbush, 1992; Kaplan & Elliott, 1995). 

. Discussion 

The purpose of this paper was to explore the use of the probability of mathematics 
proficiency as a measure of student achievement. It was hypothesized that relationships specified 
between indicators of a model of schooling may differ, depending upon which level of probability 
of proficiency is under investigation. It was also theorized that by using the five outcome levels, 
one would gain more information about the complex relationships between indicators of schoolin° 
and student learning, versus using a total mathematics achievement score as the final outcome 
measure. By breaking down the total score into skill categories one can study the relationships 
between important education indicators and student achievement at a specific skill level. Also, the 
relationships between indicators may differ across skill types. This is information that would not 
be obtained with the use of a total mathematics achievement score. 

Several interesting findings emerged. First, in developing the within-school model, 
differences in the magnitudes of the modification indices and expected parameter change statistics 
emerged, depending on which level of outcome was used. Therefore, if one uses the method of 
model modification involving either the Ml, EPC, or both (see Saris et al., 1987, Kaplan 1989, 
1990), then the modifications made to the five models may differ. Thus, different models may 



’The sample size for the multilevel analysis is N - G = 1393. 
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emerge, dependent upon which level of probability of proficiency was under investigation. This 
suggests that the relationships between the indicators in the model differ, depending upon the skill 
level being studied It is quite possible that different indicators might be required to study the 
effects of schooling on student proficiency at different skill levels. 

A related finding was that the five within-school models differed in degree of 
statistical fit. For identical models, except for the final outcome measure, results yielded 
differential values across a variety of statistical fit indexes. This offers additional evidence of the 
need for different within-school models of the probability of mathematics proficiency across skill 
levels to explain the interrelationships between indicators of student achievement and the learning 
of different skills. 

Finally, several estimated effects between indicators varied in magnitude, depending 
upon which level of outcome was under investigation. Although the differences were moderate, 
they show that the relationships between indicators of the model differ across skill levels. This 
result, combined with the differences in statistical fit of the model to the data, and the differences 
in Mis and EPCs across models, offer evidence for the need to separately develop models to 
explain student achievement for each given skill level. 

It is important to note that these skill levels are developmental, as discussed in the 
domain of cognitive psychology. One aspect of cognitive psychology related to the learning at 
specific skill levels is the development of working memory. As a person develops, he or she can 
construct more complex working memory programs which allow for quicker operations. This 
affords the retention of longer and more complex instructional subroutines (such as problem- 
solving strategies, Chi, 1977). A second aspect related to learning at specific skill levels is the 
development of long-term memory and its relation to knowledge acquisition (Famham-Diggory, 
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1990). For example, conceptual knowledge 2 starts developing at infancy and gradually becomes 
hierarchical. Procedural knowledge 3 is acquired more rapidly from birth, while analogial 
knowledge 4 does not change with development, but rather gets used increasingly more effectively. 
Therefore, with development comes the capacity for learning at higher skill levels, as well as an 
increase in one’s probability of proficiency at each skill level. 

Cognitive development affects the understanding and use of numeracy mechanics, or 
basic skills (Famham-Diggory, 1990). Basic skills consist of: (1) first order skills; and (2) second- 
order skills. First-order skills deal with small units such as addition, subtraction, multiplication, 
and division. They include the retrieving of“number facts” and calculating. These would include 
the use of simple operations, or Level 1 and Level 2 skills as defined by NCES (1988). Second- 
order skills deal with larger units and are usually considered to be the problem-solving skills, or 
Level 3, Level 4, and Level 5 skills (NCES, 1988) These involve the learning mathematical 
heuristics, or “rules-of-thumb” to help guide one in finding a correct solution (Shoenfeld, 1985). 
The heuristics become more complex with development, allowing one to solve increasingly more 
complex problems. Thus, offering evidence. that the skill levels are defined by NCES (1988) are 
hierarchical as well as developmental in nature (se e g. Seigler, 1991, regarding the development 
of academic skills). Therefore, factors affecting cognitive development may also explain some of 
the variation in student learning at specific skill levels. 

2 Conceptual knowledge is the knowledge of general forms or prototypes. 

3 Procedural knowledge is skill knowledge, or knowledge of how to do something. It is learned 
through practice. 

4 Analogical knowledge if knowledge that preserves the patterned structure of information and 
aids with the recall of other information, such as facts or concepts (Farnham-Diggory, 1990). 
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Conclusions / Caveats 

This article discussed the use of probability of proficiency scores as a means to gain 
invaluable information about how school factors can differentially affect student learning across 
skill levels. An important limitation emerges from this work. The public release files of NELS:88 
contain variables that have been recoded to preserve the confidentiality of the respondents. For 
example, many continuous variables have been converted to Likert scales. Since the metric of 
Likert scales are potentially meaningless (a few of the indicators have meaningful zero values 
wherein zero means "a lack of the characteristic being measured by that indicator"), the use of 
categorical data limits the interpretation of the coefficients relating the within and between-school 
levels. Therefore future work should involve the re-estimation of this model, either by utilizing 
the confidential files of NELS:88, which contains variables that have been suppressed from the 
public release files, as well as many variables in their original continuous form (before recoding), 
or by using another data set that contains more variables coded in a meaningful metric. 

This article offers evidence for the need to build models geared towards the study 
of student learning of specific skills. Results showed that across the five models, there are 
differences in statistical fit, differences in the Mis and EPCs, and differences in the coefficients 
relating the indicators of the model across outcome levels. Future work should focus on the 
development of individual models of the probability of mathematics proficiency for each skill 
level. It is hypothesized that the models will differ in the magnitudes of the relationships between 
indicators, in the specification of the relationships in the model, and in the indicators actually 
included in each of the models. It is possible that an indicator important for studying learning of 
one skill level may be unimportant for learning skills at a different level. Therefore, learning at 
each skill level should be studied via its own model of schooling. 
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Appendix 



The use of multilevel structural equation modeling will allow one to capture the 
multilevel, organizational nature of schooling, and the interrelationships amongst indicators within 
as well as across levels of the system (Darling-Hammond, 1 992; Oakes, 1 986; Porter, 1991, 
Willms, 1992) Studies of school systems up until this point have routinely either ignored the 
organizational structure of schools, used aggregated data, or used data from several levels of 
schooling within a uni-level model, and have therefore offered at best biased results and 
misleading conclusions (see e g. Bryk & Raudenbush, 1988; Lee & Bryk, 1989). A few of the 
earlier studies (before the development of sophisticated multilevel modeling techniques) 
recognized the problems of using aggregated data, but justify its use by acknowledging the 
difficulty of estimating and interpreting multilevel analysis in the same model (e g. Bidwell and 
Kasarda, 1975). 

There has recently been a flurry of activity in developing and estimating multilevel 
regression models of education (e g. Bryk & Raudenbush, 1992; Fitz-Gibbon, 1991; Gamoran, 
1991; Monk & King, 1 994; Lockheed & Longford, 1991; Zuzovsky & Aitkin, 1991). However. 
HLM is limited since it cannot model the structural relationships existing amongst variables within 
and between levels of the system, nor can HLM decompose the relationships between variables of 
the model into the direct, indirect, and total effects of the variables within and between levels oi 
the system. Multilevel structural equation modeling offers this information, plus it can allow for a 
measurement model to be specified, affording the inclusion of latent variables. 

For ease of discussion about multilevel structural equation modeling I will assume 
that the indicators are valid and reliable. However, I recognize that in most cases this assumption 
is unreasonable and point out that the technique can be extended to the case wherein multiple 
measures are used via the building of a multilevel measurement model (see Muthen, 1991, 1994i 
The model to be discussed below has been previously discussed in Kaplan and Elliott (in press) 
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Starting with the within-school model, it is assumed that the intercepts and means of 
the students level indicators vary across schools. It is also assumed that there exists a between- 
school model which can explain this variation. Finally, it is assumed that the slopes are fixed 
Keeping the aforementioned assumptions in mind, the within-school model can be written as 
follows. 



y. = a + B y. + e- , ( 1 ) 

where y jg is a vector of student level indicators, some of which are exogenous for the / lh student (/ 
= I,..., N) in the#** 1 school (g = I,..., G), a g is a vector of intercepts of the student level indicators 
which are assumed to vary across schools, B y is a matrix of regression coefficients relating the 
student level coefficients to each other, and s is the disturbance term for the student level 

equation. Equation (1) is also known as the structural form of the within-school model, 
representing the relationships between the indicators of the within-school model. To model the 
variation in the intercepts of the student level indicators it is useful to re-write equation ( 1 ) in its 
reduced form 



y = (I- - B ) ‘ a +(I-B ) 'e , 

J <g v y ' g v y' >g’ 



( 2 ) 



where it is assumed that (I - B y ) exists (Muthen, 1994). 

As explained above, it is assumed that there exists a between-school model that 
explains the variation in the means and intercepts of the within-school indicators across schools 
This variation can be modeled as follows: 



ERIC 



a=a+Bz+5, 

g a g g ’ 



( 3 ) 



where a is the grand mean vector across G schools, z g are school level exogenous and 
endogenous indicators, B a is a matrix of regression coefficients which relate z g to the intercepts of 
the student level indicators, and 5 g is a vector of disturbance terms. 

As explained by Muthen (1994), equations (1), (2), and (3) allow intercepts and 
means to vary as a function of school level indicators. As alluded to by Muthen (1994), and 
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explained by Kaplan & Elliott (in press), the between-school indicators z g are allowed to follow a 
separate between-school model, written as follows: 

z = x + B z + u , (4) 

where x is a vector of means and intercepts for the school level exogenous and endogenous 
indicators, B is a matrix of coefficients which relate school level indicators to each other, and u 

is a vector of disturbances. Equation (4) can be re-written in reduced form 

z g = (* - B z ) T + 0 - B z ) ' u g , (5) 

-i 

assuming (I - B^,) exists. 

From a series of substitutions using equations ( 1 ) through (5) comes the expression 
for the / ,h students score in theg 4 * 1 school, taking into account the structural relationships within 
and between schools. This model can be expressed as 

y ig = (I - B y ) 1 a + nx + n Ug + (I - B y ) 1 5 s + (I - By) ' s ig , (6) 

where n = (I - B )' B a (I - B z ) is a matrix containing regression coefficients relating school level 
exogenous variables to student level endogenous variables, taking into account the between- 
school and within-school structures, n is also known as a multilevel total effects matrix (Kaplan 
& Elliott, in press). The expected value of y ig can be written as 

E(yi g ) = (I - B y ) ‘a + rix (7) 

and the variance can be written as 

V(y jg ) = ny u n + (I - By)' ■’ T5 (I - By)' V (I - By)' 1 y C (I - By)' ^ (8) 

where ¥„ = Var(a g ), 4^5 = Var(8 g ), and 4 / c = Var(s g ). This leads to the expression for the 
between - and within-group covariance matrices, 

Ep= ny u ri' +(I -ByX’TsCI - By)' - 1 (9) 

and 

I W = (I- By)- , 'Fe(I-By)'-' (10) 
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It is evident form equation (1 5) that the expected value of the i student’s scores in 
the g* school is the weighted sum of the grand mean vector of student level variables (a) and the 
grand mean vector of the school level variables (t). 
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Table 1 Within-School Model Indicators 
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MATH 1 Probability of mathematics proficiency at Level 2 as continuous, ranging from 0- 1 
defined by "simple arithmetical operations on whole 
numbers (single step oeprations which rely on rote memory." 
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Table 2 Betwecn-School Model Indicators 
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Tabic 3 



Descriptive Statistics for Within-School Model 
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Mean 


St Dev 


Skew 


Kurtosis 


Min 


Max 


MATH 1 


0.949 


0.136 


-3.750 


14.928 


0.03 


1.00 


MATH2 


0.735 


0.389 


-1.066 


-0.642 


0.00 


1.00 


MATH3 


0.539 


0.458 


-0.151 


-1.862 


0.00 


1.00 


MATH4 


0.255 


0.358 


1.1 12 


-0.420 


0.00 


1.00 


MATH 5 


0.006 


0.029 


7.645 


70.422 


0.00 


0.03 


HOME- 

WORK 


2.71 1 


1.642 


1.761 


6.218 


0.00 


14.00 


PRIOR 


3.078 


0.929 


-0.865 


0.338 


0.00 


4.00 


CLUB 


0.369 


0.516 


0.91 1 


-0.394 


0.00 


2.00 


PROBLEM 


2.366 


0.835 


-1.211 


0.697 


0.00 


3.00 


EFFORT 


3.227 


1.092 


-1.518 


1.555 


0.00 


4.00 


GRADING 


9.081 


3.079 


-0.207 


-0.581 


0.00 


15.00 


HANDS 


1.639 


1.191 


0.527 


0.176 


0.00 


6.00 


YRSEXP 


5.299 


2.732 


-0.176 


-1.274 


0.00 


9.00 


ENRICH 


0.846 


0.361 


-1.921 


1.692 


0.00 


1.00 


CL AS SIZE 


23.245 


7.090 


1.173 


9.165 


1.00 


90.00 


REQUIRED 0.428 


0.495 


0.293 


-1.912 


0.00 


1.00 



Table 4 


Fit Statistics for Within-School Model 








K 2 ' 


df 


NNFI a 


RMSEA b 


P c 


Level 1 


. * 


302.03 


31 


0.713 


0.042 


0.998 


Level 2 




363.41 


31 


0.713 


0.047 


0.862 


Level 3 (Initial) 


827.48 


37 


0.484 


0.066 


0.000 


Level 3 (Final) 


490.74 


31 


0.642 


0.055 


0.020 


Level 4 




481.60 


31 


0.618 


0.055 


0.032 


Level 5 




315.77 


31 


0.654 


0.044 


0.992 



a NNFI = Non-Normed Fit Index 

b RMSEA = Root Mean Squared Error of Approximation 
c p<.05 refers to the probability of the value of p being less than .05 
* P < .05 for all Levels 





Probability of Proficiency 

Table 5 Modification Indexes and Expected Parameter Change Statistics 
for Within-School Models, Developed with Level 3 as Outcome 



Probability of Proficiency Level 



Freed Path Level I Level 2 Level 3 Level 4 Level 5 

Ml EPC Ml EPC Ml EPC Ml LPC Ml LPC 

HANDS- MATH 0.12 0.00 7.04 0.04 94.79 0. 1 3 9 1.56 0.14 16.82 0.06 



HANDS - 

PROBLEM 80.0 20.13 80.02 0.13 80.02 0.1.3 80.02 0.13 '80.02 0.13 



PRIOR - 

HOMEWORK 47.43 0.10 47.43 0.10 47.54 0.10 47.43 0.10 47.43 0.10 
REQUIRED - 

HOMEWORK 23.96 0.07 23.96 0.07 23.95 0.07 23.96 0.07 23.96 0.07 



HOMEWORK- 

CLUB 42.78 0.09 36.87 0.09 42.78 0.10 36.78 0.05 42.78 0,09 

PROBLEM - 

MATH 45.73 0.09 82.01 0.13 40.99 0.08 16.07 0.03 0.59 0.0 1 




3 



9 
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Table 6 Direct, Indirect, and Total Effects of Within School Model with 
Math 1 as Outcome: Final Model 







MATH 1 


HOMEWORK CLUB 


PROBLEM 


EFFORT 


GRADING 


HANDS 


HOMEWORK 


direct 


0.062* 





0.095* 


. ... 


.. 




— 




indrect 


0.007* 






— - 


— 


----- 






total 


0.069* 




0.095* 


..... 








CLUB 


direct 


0.072* 








-. — ' 








indirect 






----- 


----- 










total 


0.072* 




----- 








----- 


PROBLEM 


direct 


0.096* 


0.077* 


.. 


-. — 


0.229* 


-. — 






indirect 


0.009* 


0.033* 


0.022* 






----- 






total 


0.104* 


0.11 1* 


0.022* 


- — 


0.229* 






EFFORT 


direct 


-. — 


0.146* 


0.051* 












indirect 


0.014* 


— 


0.014* 












total 


.0.014* 


0.146* 


0.065* 










GRADING 


direct 










-0.024 


0.035* 









indirect 


-0.002 


0.003 


0.002 


— 


-0.005 








total 


-0.002 


0.003 


0.002 


-0.024 


0.030* 






HANDS 


direct 


-0.016 






0 129* 


0.082* 








indirect 


0.015* 


0.026* 


0.008* 




0.030* 








total 


-0.001 


0.026* 


0.008* 


0.129* 


0.111* 






YRSEXP 


direct 


-. — 


-. — 


— 


-. — 


— 


-0.042* 


-0.029* 




indirect 


0.000 


-0.001* 


-0.000* 


-0.003 


-0.004* 








total 


0.000 


-0.001* 


-0.000* 


-0.003 


-0.004* 


-0.042* 


-0.029* 


ENRICH 


direct 


-. — 




-. — 




-. — 


0.037* 


0.098* 




indirect 


-0.000 


0.003* 


0.001* 


0.012* 


0.012* 








total 


-0.000 


-0.003* 


0.001 * 


0.012* 


0.012* 


0.037* 


0.098* 


SIZE 


direct 


— 




— 


-. — 


— 


0.092* 


-0.069* 




indirect 


-0.000 


-0.002* 


-0.000* 


-0.011* 


-0.005* 


..... 






total 


-0.000 


-0.002* 


-0.000* 


-0.011* 


-0.005* 


0.092* 


-0.069* 


REQUIRED 


direct 


-. — 


0.071* 




..... 


0.059* 


..... 






indirect 


0.006* 


0.009* 


0.011* 


- — 










total 


0.006* 


0.079* 


0.011* 


— 


0.059* 






PRIOR 


direct 


0.234* 


0.086* 


0.134* 





0.050* 










indirect 


0.016* 


0.007* 


0.011* 












total 


0.250* 


0.093* 


0.145* 




0.050* 








• E< -05 
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Table 7 
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Direct, Indirect, and Total Effects of Within School Model with 
Math 2 as Outcome: Final Model 








MATH2 


HOMEWORK CLUB 


PROBLEM 


EFFORT 


GRADING 


HANDS 


HOMEWORK 


direct 


0.067* 




0.095* 






.... 







indrcct 


0.006* 






- — 






----- 




total 


0.072* 


----- 


0.095* 


----- 


----- 


- — 




CLUB 


direct 


0.062* 




















indirect 






----- 


----- 






..... 




total 


0.062* 


----- 


----- 










PROBLEM 


direct 


0.100* 


0.077* 


-. — 


- — 


0.229* 


-. — 


-. — 




indirect 


0.009* 


0.033* 


0.022* 












total 


0.109* 


0.111* 


0.022* 




0.229* 






EFFORT 


direct 




0.146* 


0.051* 








-. — 


- — 




indirect 


0.014* 




0.014* 












total 


0.014* 


0.146* 


0.065* 










GRADING 


direct 









-0.024 


0.035* 








indirect 


-0.002 


0.003 


0.002 




-0.005 


- — 


- — 




total 


-0.002 


0.003 


0.002 


-0.024 


0.029* 


— 




HANDS 


direct 


0.025 


— 


-. — 


0.129* 


0.082* 










indirect 


0.015* 


0.026* 


0.008* 




0.030* 








total 


0.040* 


0.026* 


0.008* 


0.129* 


0.111* 






YRSEXP 


direct 


-. — 


-. — 


-. — 


-. — 


.. 


-0.042* 


-0.029* 




indirect 


0.001 


-0.001* 


-0.000* 


-0.003 


-0.004* 








total 


-0.001 


-0.001* 


-0.000* 


-0.003 


-0.004* 


-0.042* 


-0.029* 


ENRICH 


direct 


-. — 














0.037* 


0.098* 




indirect 


0.004* 


0.003* 


0.001* 


0.012* 


0.012* 








total 


0.004* 


0.003* 


0.001* 


0.012* 


0.012* 


0.037* 


0.098* 


SIZE 


direct 


— 




-. — 


— 


-. — 


0.092* 


-0.069* 




indirect 


-0.003* 


-0.002* 


-0.000* 


-0.011* 


-0.005* 




-. — 




total 


-0.003* 


-0.002* 


-0.000* 


-0.011* 


-0.005* 


0.092* 


-0.069* 


REQUIRED 


direct 


— 


0.071* 


— 




0.058* 





_ — 




indirect 


0.006* 


0.009* 


0.01 1* 












total 


0.006* 


0.079* 


0.011* 




0.058* 






PRIOR 


direct 


0.341* 


0.086* 


0.134* 


-. — 


o 

o 

C-A 

O 

* 










indirect 


0.015* 


0.007* 


0.011* 












total 


0.356* 


0.093* 


0.145* 




0.050* 






* g < .05 


j.iSAJ 


iUyk ' 


i --in \ T 
' ■ ; -< 
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Table 8 Direct. Indirect, and Total effects of Within School Model with 
Math 3 as Outcome: Final Model 
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MATH 3 


HOMEWORK CLUB 


PROBLEM 


KPFORT 


GRADING 


1 IANDS 


HOMEWORK 


direct 


* 

e'- 

en 

o 

O 




0.095* 












indrcct 


0.006* 






— 




----- 






total 


0.062* 




o 

o 




----- 




----- 


CLUB 


direct 


0.061* 
















indirect 




..... 








----- 






total 


0.061* 














PROBLEM 


direct 


0.086* 


0.077* 


- — 


_. 


0.226* 








indirect 


0.008* 


0.033* 


0.022* 




— 




..... 




total 


0.094* 


0.110* 


0.022* 


----- 


0.226* 




----- 


EFFORT 


direct 


-. — 


0.146* 


0.051* 


.. 












indirect 


0.012* 




0.014* 


-. — 






-. — 




total 


0.012* 


0.146* 


0.065* 










GRADING 


direct 









-0.024 


m 

O 

O 








indirect 


-0.002 


0.003 


0.002 




-0.005 








total 


-0.002 


0.003 


0.002 


-0.024 


0.030* 


. ... 




HANDS 


direct 


0.121* 








0.129* 


0.093* 










indirect 


0.013* 


0.028* 


0.009* 


-. — 


0.029* 


-. — 






total 


0.134* 


0.028* 


0.009* 


0.129* 


0.122* 






YRSEXP 


direct 


-. — 


— 





_. 





-0.042* 


-0.029* 




indirect 


0.004 


-0.001* 


-0.000* 


-0.003 


-0.005* 








total 


-0.004 


-0.001* 


-0.000* 


-0.003 


-0.005* 


-0.042* 


-0.029* 


ENRICH 


direct 


. — 


.. 





_. 





o 

o 


0.098* 




indirect 


0.013* 


0.003* 


0.001* 


0.012* 


0.013* 


.. — 


-. — 




total 


0.013* 


0.003* 


0.001* 


0.012* 


0.013* 


0.037* 


0.098* 


SIZE 


direct 


-. — 




-. — 


_. 





0.092* 


-0.069* 




indirect 


-0.009* 


-0.002* 


-0.000* 


-0.01 1* 


-0.006* 


-. — 


-. — 




total 


-0.009* 


-0.002* 


-0.000* 


-0 01 1 * 


-0.006*. 


0.092* 


-0.069* 


REQUIRED 


direct 


— 


0.071* 


-. — 





0.054* 










indirect 


* 

CO 

o 

O 

O 


0.008* 


0.010* 


..... 






-. — 




total 


0.005* 


0.079* 


0.010* 




o 

o 




- — 


PRIOR 


direct 


0.366* 


0.086* 


0.134* 




0.067* 





- 




indirect 


0.014* 


0.010* 


0.013* 


■ 


-. — 


-. — 


— . 




total 


0.381* 


0.096* 


.0.147* 




0.067* 


-. — .— 
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Table 9 Direct, Indirect, and Total Effects of Within School Model with 
Math 4 as Outcome: Final Model 







MAT! 14 


HOMEWORK CLUB 


PROBLEM 


REPORT 


GRADING 


HANDS 


HOMEWORK 


direef 


0.047* 




0.095* 












indrcct 


0.007* 






..... 




----- 






total 


0.054* 




0.095* 


----- 


- — 


----- 




CLUB 


direct 


0.074* 




..... 


..... 











indirect 


..... 


..... 


----- 






----- 






total 


0.074* 


..... 


----- 


----- 


----- 


----- 


— 


PROBLEM 


direct 


0.055* 


0.077* 








0.229* 




-. — 




indirect 


0.007* 


0.033* 


0.022* 




-. — 








total 


0.062* 


0. Ill * 


0.022* 


----- 


0.229* 






EF70RT 


direct 


— 


0.146* 


0.051* 
















indirect 


0.012* 




0.014* 


- — 










total 


0.012* 


0.146* 


0.065* 










GRADING 


direct 


..... 






-0.024 


0.035* 








indirect 


-0.001 


0.003 


0.002 




-0.005 








total 


-0.001 


0.003 


0.002 


-0.024 


0.029* 




— 


HANDS 


direct 


0.093* 








0.129* 


0.082* 










indirect 


0.009* 


0.026* 


0.008* 


-. — 


0.030* 




— 




total 


0.102* 


0.026* 


0.008* 


0.129* 


0.111* 






YRSEXP 


direct 


— 


-. — 











-0.042* 


-0.029* 




indirect 


-0.003 


-0.001* 


-0.000* 


-0.003 


-0.004* 


-. — 


-. — 




total 


-0.003 


-0.00 1* 


-0.000* 


-0.003 


-0.004* 


-0.042* 


-0.029* 


ENRICH 


direct 


— 














0.037* 


0.098* 




indirect 


0.010* 


0.003* 


0.001* 


0.012* 


0.012* 


-. — 


-. — 




total 


0.010* 


0.003* 


0.001* 


0.012* 


0.012* 


0.037* 


0.098* 


SIZE 


direct 


— 




-. — 







0.092* 


-0.069* 




indirect 


-0.007* 


-0.002* 


-0.000* 


-0.011* 


-0.005* 


-. — 


-. — 




total 


-0.007* 


-0.002* 


-0.000* 


-0.011* 


-0.005* 


0.092* 


-0.069* 


REQUIRED 


direct 


-. — 


0.071* 


— 


.. 


0.058* 










indirect 


0.005* 


0.009* 


0.011* 


— 






-. — 




total 


0.005* 


0.079* 


0.011* 


- — 


0.058* 


----- 




PRIOR 


direct 


0.319* 


0.086* 


0.134* 





0.050* 










indirect 


0.015* 


0.007* 


0.011* 


-. — 


-. — 










total 


0.335* 


0.093* 


0.145* 


-. — 


0.050* 




-. — 



’ E< 05 
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Table 10 Direct, Indirect, and Total Effects of Within School Model with 
Math 5 as Outcome: Final Model 







MATHS 


HOMEWORK 


CLUB 


PROBLEM 


EFFORT 


GRAD1NC 


; HANDS 


HOMEWORK 


direct 


0.018 




0.095* 




.... 










indrcct 


0.007* 




----- 




----- 


----- 






total 


0.025 




0.095* 




----- 






CLUB 


direct 


0.071* 

















. ... 




indirect 






----- 


- — 


----- 


----- 






total 


0.071* 




- — 




----- 


----- 


----- 


PROBLEM 


direct 


0.01 1 


0.077* 


-. — 


-. — 


0.229* 


-. — 


- — 




indirect 


0.004* 


0.033* 


0.022* 






----- 






total 


0.015 


0.1 II* 


0.022* 




0.229* 


----- 




EFFORT 


direct 


-• — 


0.146* 


0.051* 


-. — 


-. — 


-. — 


-. — 




indirect 


0.007* 




0.014* 




----- 








total 


0.007* 


0.146* 


0.065* 




----- 






GRADING 


direct 





_ „ 


- r - - 


-0.024 


0.035* 










indirect 


-0.000 


0.003 


0.002 




-0.005 








total 


-0.000 


0.003 


0.002 


-0.024 


0.029* 






HANDS 


direct 


0.058* 


-. — 


-. — 


0.129* 


0.082* 




-. — 




indirect 


0.003* 


0.026* 


0.008* 




0.030* 


----- 






total 


0.060* 


0.026* 


0.008* 


0.129* 


0.11 1* 


----- 




YRSEXP 


direct 


— 


— 


-. — 


— 


-. — 


-0.042* 


-0.029* 




indirect 


-0.002 


-0.001* 


-0.000* 


-0.003 


-0.004* 








total 


-0.002 


-0.001* 


-0.000* 


-0.003 


-0.004* 


-0.042* 


-0.029* 


ENRICH 


direct 


— 














0.037* 


0.098* 




indirect 


0.006* 


0.003* 


0.001* 


0.012* 


0.012* 








total 


0.006* 


0.003* 


0.001* 


0.012* 


0.012* 


0.037* 


0.098* 


SIZE 


direct 


— 




-. — 








0.092* 


-0.069* 




indirect 


-0.004* 


-0.002* 


-0.000* 


-0.011* 


-0.005* 








total 


-0.004* 


-0.002* 


-0.000* 


-0.01 1* 


-0.005* 


0.092* 


-0.069* 


REQUIRED 


direct 


-. — 


0.071* 


-. — 


— 


0.058* 


-. — 


-. — 




indirect 


0.002 


0.009* 


0.011* 








-. — 




total 


0.002 


0.079* 


0.01 1* 




0.058* 






PRIOR 


direct 


0.141* 


0.086* 


0.134* 


-. — 


0.050* 


-. — 







indirect 


0.012* 


0.007* 


0.011* 




-. — 








total 


0.153* 


0.093* 


0.145* 




0 . 050 * 







O *E < 05 




44 BEST COPY AVAILABLE 



Probability of Proficiency 40 



Table 1 1 


Intraclass Correlations 


Indicator 


Intraclass Correlation 


MATH 1 


.109 


MATH 2 


.165 


MATH 3 


.172 


MATH 4 


.151 


MATH 5 


.030 


HOMEWORK .067 


PRIOR 


.084 


CLUB 


.071 


PROBLEM 


.052 


EFFORT 


.032 


GRADING 


.288 


HANDS 


.228 


YRSEXP 


.370 


ENRICH 


.304 


SIZE 


.395 


REQUIRED 


.073 



Table 12 Descriptive Statistics for Between-School Model 





Mean 


St Dev 


Skew 


Kurtosis 


Min 


Max 


PRESS 


4.036 


0.750 


-0.323 


-0.465 


2.000 


5.000 


ADVMATH 


1.441 


1.943 


2.512 


8.818 


0.000 


12.000 


STAFDEV 


3.072 


0.794 


-0.684 


0.758 


0.000 


4.000 


DROPOUT 


0.550 


0.500 


-0.202 


-1.996 


0.000 


1.000 


GROUPING 0.820 


0.386 


-1.687 


0.785 


0.000 


1.000 


MTEACHR 


2.991 


0.910 


0.682 


-0.311 


2.000 


5.000 


SALARY 


4.090 


1.203 


0.175 


-0.766 


2.000 


7.000 


LUNCH 


1.396 


0.730 


0.082 


-0.266 


0.000 


3.Q00 




liciA 



45 



Probability of Proficiency 41 



Tabic 13 Direct, Indirect, and Total Effects of Bctwccn-Scliool Model 







PRESS 


MTEACHR 


STAFDEV 


ADVMATH 


MTEACHR 


direct 


0.052 












indirect 












total 


0.052 








STAFDEV 


direct 


0 . 261 * 










indirect 


0.003 










total 


0 . 264 * 


0.058 






ADVMATH 


direct 


0.101 


0.230 








indirect 


0.012 










total 


0.113 


0 . 230 * 






LUNCH 


direct 


- 0.106 




- 0.124 


- 0.25 r 




indirect 


- 0.061 


- 0 . 065 * 








total 


- 0.167 


- 0 . 065 * 


- 0.124 


-0.25 r 


GROUPING 


direct 




0.098 








indirect 


0.005 










total 


0.005 


0.098 






DROPOUT 


direct 






0 . L 60 






indirect 


0.042 


0.009 








total 


0.042 


0.009 


0.160 




SALARY 


direct 




0 . 388 * 




0.059 




indirect 


0.027 


0 . 014 * 








total 


0.027 


0 . 402 * 




0.059 



*£< .05 
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Table 14 Fit statistics for the Multilevel Model 





l 2 


df 


NNFI 3 


RMSEA 1 ’ 


P C 


Level 1 


409.99* 


347 


0.902 


0.011 


1.000 


Level 2 


447.68’ 


347 


0.876 


0.014 


1.000 


Level 3 


456.89* 


347 


0.868 


0.015 


1.000 


Level 4 


450.12* 


347 


0.865 


0.014 


1.000 


Level 5 


392.12** 


347 


0.914 


0.009 


1.000 



3 NNFI = Non-Normed Fit Index 

b RMSEA = Root Mean Squared Error of Approximation 

c p<.05 refers to the probability of the value of the RMSEA being less than .05 

* p < .05 

** P = .048 



Table 15 Standardized Regression Coefficients of the Within-School 
Intercepts on PRESS 



Intercept 


Level l 


Level 2 


PRESS 
Level 3 


Level 4 


Level 5 


MATH 


0.016 


0.053 


0.078 


0.058 


0.003 


HOMEWORK 


-0.091 


-0.091 


-0.091 


-0.091 


-0.091 


CLUB 


-0.021 


-0.021 


-0.021 


-0.021 


-0.021 


PROBLEM 


0.026 


0.026 


0.026 


0.026 


0.026 


EFFORT 


-0.008 


-0.008 


-0.008 


-0.008 


-0.008 


GRADING 


-0.160 


-0.160 


-0.160 


-0.160 


-0.160 


HANDS 


0.010 


0.010 


0.010 


0.010 


0.010 


YRSEXP* 


-.— 









-.— 


ENRICH 


0.029 


0.029 


0.029 


0.029 


0.029 


SIZE* 


-.--- 












REQUIRE* 













PRIOR* 




----- 










Indicator is exogenous to the model 
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